Skip to content

fix(server): repeated capture timeouts no longer delay turn starts - #29196

Open
voltcrash wants to merge 1 commit into
pingdotgg:mainfrom
voltcrash:voltcrash/baseline-capture-timeout
Open

voltcrash wants to merge 1 commit into
pingdotgg:mainfrom
voltcrash:voltcrash/baseline-capture-timeout

Conversation

@voltcrash

@voltcrash voltcrash commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Repeated checkpoint timeouts make each new provider turn wait about 30 seconds before starting without a baseline. After two consecutive capture timeouts in a workspace, this skips missing-baseline retries. Every post-turn capture still runs as the recovery probe, and a successful capture immediately clears the streak. Existing refs, pre-provider snapshot ordering, and restore validation stay intact.

Addresses #28996.

Verification:

  • The focused regression failed before the fix: a third baseline invoked capture when two attempts were expected. It passes after the fix, including workspace isolation, post-turn retries, and recovery on success.
  • vp run check:changed passes format, lint, knip, generated GraphQL checks, affected typechecks, and adjacent tests. An additional 68 checkpoint capture, rollback, and run-execution tests pass.
  • Real-Git runtime check, with VcsProcessTimeoutError injected only at the checkpoint git add boundary (no actual 30-second wait):
first baseline: Failure stage attempts: 1
post-turn checkpoint: error stage attempts: 2
next baseline: Success stage attempts: 2
uncaptured baseline: missing restore: Failure
healthy post-turn: ready stage attempts: 3
new baseline after recovery: Success stage attempts: 4
restored successful checkpoint: "recovered snapshot\n"

The next baseline succeeds by skipping capture, logs that skip, and remains missing rather than advertising a restorable snapshot. Native Windows and the reporter's workspace were not tested.

Trade-off: after repair, one pre-turn baseline can remain missing until the next successful post-turn capture. Initial and post-turn timeouts keep their existing budget. Timeout streaks are in memory and bounded to 128 workspaces; success or a non-timeout capture failure clears them. No changes to steer handling, Git temporary-object cleanup, or forge CLI concurrency.

Reviewed with review-loop: 2 rounds with Claude Opus 5.5 on high. Reviewer verdict: approved.

Created with GPT-6.1 Sol (Extra High) in the Codex harness through T3 Code.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Nov 2, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at b7208c0

Macroscope's review found this PR approvable — Repeated capture timeouts now suppress baseline retries for that workspace while post-turn captures continue checking for recovery. Healthy captures remain unchanged, and successful recovery clears the temporary backoff.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

CheckpointServiceV2 tracks consecutive checkpoint capture timeouts per workspace. After two timeouts, it skips baseline captures for that workspace. Successful captures and non-timeout failures clear the count.

Changes: Checkpoint capture timeout backoff

Layer / File(s) Summary
Per-workspace timeout tracking
apps/server/src/orchestration-v2/CheckpointService.ts
A cache tracks capture outcomes per workspace. Successful captures and non-timeout failures clear the count. Timeout failures increment it.
Capture paths and retry behavior
apps/server/src/orchestration-v2/CheckpointService.ts, apps/server/src/orchestration-v2/CheckpointService.test.ts
Baseline capture skips a workspace after at least two recorded timeouts. Regular capture uses the shared wrapper and preserves its missing and error results. The test checks that backoff is workspace-scoped and that a successful post-turn capture clears it.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Possibly related issues

  • Issue 28996: The PR skips later baseline captures after repeated timeouts, but does not address the initial timeout or its duration.
  • Issue 17825: The PR skips baseline capture after repeated timeouts, but does not address an indefinite hang.
  • Issue 26227: The PR skips baseline captures after repeated timeouts and continues post-turn captures, but does not address leftover directories or capture convergence.

Possibly related PRs

  • pingdotgg/t3code#31657: It extends baseline and post-turn capture to attached repository worktrees, the same capture paths affected by this PR.
  • pingdotgg/t3code#30972: It changes checkpoint event construction when a materialized baseline is not ready, a condition that can follow capture timeouts.
  • pingdotgg/t3code#26254: It invokes baseline and post-turn capture across repository scopes, the capture operations whose retry behavior this PR changes.

Suggested reviewers: jonasfocus

Merge Risk: Minimal · up to ea1e0

No identified issue needs resolution before merge; normal checks remain appropriate.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly summarizes the timeout backoff change and its effect on turn starts.
Description check Passed The description is mostly complete. It explains the problem, fix, trade-off, and verification results. It does not use the required section headings or establish whether the linked issue was triaged o…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description is mostly complete. It explains the problem, fix, trade-off, and verification results. It does not use the required section headings or establish whether the linked issue was triaged or the scope was approved.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Usage-based review receipt

Note

This review was completed with usage-based billing: files reviewed beyond your plan's included limits are billed at $0.25/file. View usage-based billing.

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between b7208c0 and ea1e0ed.

Files selected for processing: 2

  • apps/server/src/orchestration-v2/CheckpointService.test.ts
  • apps/server/src/orchestration-v2/CheckpointService.ts

Limit details: You’ve used all 10 included reviews currently available.

@maria-rcks

maria-rcks commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Written by gpt-6.1-sol on behalf of Maria

Verified in a real dev environment

verified as-is at b7208c06beb1e4306f827fad807dae83303d89b8 on this linux boat vm. installed and ran github/main (30084f0018) in a separate worktree first, then the exact pr head. only one background dev server ran at a time. drove the actual web client with repo playwright and chrome, using real codex gpt-6.1-sol turns through the local proxy.

codex created a real 3 gib untracked binary in the app project. actual git staging exceeded the existing 30-second budget; no mocks, replay, or injected errors were used.

  • main reproduced repeated baseline delays: startRootRun took 30,215.8 ms and 30,209.3 ms. codex started about 30,173 ms into each run.
  • the pr initial baseline and post-turn captures still timed out. after those two failures, the next baseline was skipped: startRootRun took 53.8 ms, with codex starting at +32.9 ms and returning its answer in the client.
  • the next post-turn capture still ran and timed out, proving recovery probes remain active.
  • codex excluded the large binary through .git/info/exclude. post-turn capture then succeeded in 82.0 ms. a fresh thread captured a new baseline successfully in 80.6 ms, proving recovery cleared the backoff.
  • healthy file changes produced the expected client diff. in an app-created isolated worktree, two codex turns changed note.txt; edit from here > revert files too restored the first snapshot and rewound the conversation. the shared-checkout restore guard also refused restoration and preserved the file.

63 focused checkpoint/capture/run-execution tests passed on blacksmith. vp run check:changed 47f2c8d29e981c7723467efe137ee19e91df924e passed format, lint, knip, graphql, affected typechecks, and adjacent tests after supplying missing worker git history. both independent scoped code reviews found no blockers. no code changes were needed.

main recording, elapsed time visible:

main repeats the baseline timeout before codex starts

pr recording, same workspace and prompt after two timeouts:

pr skips the baseline retry and returns the codex answer

runtime command transcript and measured spans. both recordings were inspected, played through, and checked for seeking after upload. they capture the web interaction at one frame per second; timing numbers above come from the real server spans.

Not verified

native electron, ios/android, windows/macos, the exact reporter workspace, other providers, and the original follow-up-steer symptom. this pr preserves the initial and post-turn timeout budgets; it does not eliminate those waits. the linux web/codex path, recovery, and neighboring checkpoint behavior were exercised directly.

@voltcrash
voltcrash force-pushed the voltcrash/baseline-capture-timeout branch from b7208c0 to ea1e0ed Compare November 7, 2026 09:22
@ElliotDrel

Copy link
Copy Markdown
Contributor

Note

🤖 gpt-6.1-sol responding on behalf of Elliot

Exact-head hosted checks and CodeRabbit review are green at ea1e0ed518d8f14d8437c080c7baaf33757c43dd, with no actionable code findings. The remaining description observation is accurate: the live body has prose plus Verification: rather than the repository's section headings.

@voltcrash A description-only adjustment can preserve all current content: put the repeated-delay explanation under ## Problem, the two-timeout skip/recovery policy and trade-off under ## Change, add ## Scope and approval, and change Verification: to ## Verification. A factual scope paragraph is:

Addresses #28996 through a focused correction to repeated baseline retries after demonstrated checkpoint timeouts. Initial/post-turn budgets, capture ordering, and restore validation remain unchanged. No maintainer direction approval is claimed; maintainers determine eligibility under the focused obvious-bug exception.

Keep the current model/harness attribution and evidence limits. No implementation or additional test change is requested by this description observation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants